Skip to content

[action] [PR:709] [module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS#726

Merged
mssonicbld merged 1 commit into
sonic-net:202605from
mssonicbld:cherry/202605/709
Jul 24, 2026
Merged

[action] [PR:709] [module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS#726
mssonicbld merged 1 commit into
sonic-net:202605from
mssonicbld:cherry/202605/709

Conversation

@mssonicbld

Copy link
Copy Markdown
Collaborator

Description

Add a "recovery": 600 (10 min) entry to ModuleBase._TRANSITION_TIMEOUT_DEFAULTS.

set_module_state_transition() validates the requested transition_type against the keys of _TRANSITION_TIMEOUT_DEFAULTS. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via set_module_state_transition(name, 'recovery'); because 'recovery' was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.

Motivation and Context

The 'recovery' transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in _TRANSITION_TIMEOUT_DEFAULTS, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.

recovery has no platform.json override (only startup/shutdown/reboot/halt_services are overridable in _load_transition_timeouts()), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared in finally; 600 s is ample headroom.

How Has This Been Tested?

  • Unit tests: updated the timeout-loading tests in tests/module_base_test.py to expect the new key. python3 -m pytest tests/module_base_test.py115 passed.
  • On-device with both master and 202605 image: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to ready_status=true.

Additional Information (Optional)

Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the 'recovery' transition this default enables.

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

…DEFAULTS

#### Description

Add a `"recovery": 600` (10 min) entry to `ModuleBase._TRANSITION_TIMEOUT_DEFAULTS`.

`set_module_state_transition()` validates the requested `transition_type` against the keys of `_TRANSITION_TIMEOUT_DEFAULTS`. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via `set_module_state_transition(name, 'recovery')`; because `'recovery'` was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.

#### Motivation and Context

The `'recovery'` transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in `_TRANSITION_TIMEOUT_DEFAULTS`, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.

`recovery` has no `platform.json` override (only `startup`/`shutdown`/`reboot`/`halt_services` are overridable in `_load_transition_timeouts()`), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared in `finally`; 600 s is ample headroom.

#### How Has This Been Tested?

- Unit tests: updated the timeout-loading tests in `tests/module_base_test.py` to expect the new key. `python3 -m pytest tests/module_base_test.py` → **115 passed**.
- On-device with both master and 202605 image: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to `ready_status=true`.

#### Additional Information (Optional)

Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the `'recovery'` transition this default enables.

Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Original PR: #709

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@mssonicbld
mssonicbld merged commit 58def69 into sonic-net:202605 Jul 24, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant